1. Field of the Invention
The present invention relates to the field of product testing. More specifically, the present invention relates to testing software for the automatic testing of products, which is also extensible to allow for new and modified products.
2. The Background
Development and testing of computer-related products, including software, computer chips, network routers, etc. is a costly and time consuming process. Generally, computer-related products are tested using one or more software test routines, where test input is fed to the product and the output is examined to determine if the product performs as expected. Testing software, however, has become more advanced in recent years and now may allow testers to perform extremely complex tests of their products, including things like alteration of internal settings, simulation of random external events, and formatting of output results.
One problem with testing software, however, is that it generally must be designed specifically for the product being tested. Thus, a product manufacturer must first spend time and money to create a product, then spend more time and money to create test software to properly test the product. Since many manufacturers produce multiple products, creating testing software for each of these individual products can be a daunting task, sometimes requiring a special division within the manufacturing company set aside solely to write software used to test products.
An additional problem with testing software is that it often must be run manually. The testing software needs to be executed on specific modules or portions of the product, and therefore a human tester must xe2x80x9cfeedxe2x80x9d the software into the product being tested, either by manually executing the software on the product or by providing inputs to a testing program so that it may determine which test to run and on which portion of the product. This can be very time consuming for a tester.
In the past, a modest amount of automation has been added to this process, generally in the form of batch files, where a user specifies a list of test files to be fed as input to a testing program and then a computer takes over and performs the tests with no further user interaction. These attempts at automation have suffered, however, from non-scalability (entirely new software must be created if products are redesigned, upgraded, or changed), inability to have user extendibility (generally only a single user or department is familiar with how the test software is designed, so other users in other departments cannot utilize the same software or even modify it for their needs, and inability to properly maintain reliability (again, since only single user or department are familiar with the test program and they are specifically designed for single products, any alteration of any of these facets creates instability in the program which must itself be tested before using it to test a product).
Additionally, as stated earlier, many manufacturers now have entire divisions within their company dedicated solely to test products. It is commonplace for multiple computers (hosts) to be used in testing, and for more than one of the users to be testing products at the same time. Batch files alone cannot aid in the management of these multiple processes spread over multiple machines.
What is needed is a system which allows for an automation environment for testing software which allows for scalability, user extendibility, and maintainability, as well as the management of multiple simultaneous processes.
Extensible automated testing software provides reliability, user extendibility, scalability, and multiple simultaneous testing support through the use of modules which they user may employ to set up and run test scripts. One or more job files are passed to an execution harness, which then starts a System Runner Process on a host specified in the one or more job files if one has not already been started. A connection is then made between the execution harness and the system runner process and one or more Test Runner Processes are spawned. These Test Runner Processes may be spread out over various hosts. Each of these Test Runner processes calls procedures to execute one test script at a time. Since multiple instances of the software may be run simultaneously, this allows the software to properly manage multiple tests running on multiple hosts being executed by multiple users.